﻿This manual is divided in 3 sections -

(A) System Files - Description
(B) Output Files - Description
(C) Training - An overview

---------------------------------------
SECTION (A) System Files - Description
---------------------------------------

- All the system files are placed in "ROOT\Work" directory and sub-directories.
- These files can mainly be categorized as:
	a. Trained models
	b. Moses decoder
	c. Driver program

a. The directory "ROOT\Work\trained_models" contains 6 trained models, namely -
	1. English syllabification system
	2. English to Hindi baseline transliteration system
	3. English to Hindi syllable-based transliteration system
	4. Hindi syllabification system
	5. Hindi to English baseline transliteration system
	6. Hindi to English syllable-based transliteration system
	
	These models are trained from a parallel corpus of 22,400 names. This corpus is available in
	"ROOT\Work\training-corpora.xls" file.
	
For each of the above 6 systems, there are 3 files in the directory. For the purpose of illustrations,
we'll talk about English to Hindi syllable-based transliteration system here -
	1. Language Model file ("en-hi-translit.lm") - A language model assigns a probability to a sequence of
	   'n' words (syllables in our case). The language model is created on the target side (Hi in this case).
	2. Phrase-translation table ("en-hi-translit.gz") - An entry of 
		kit ||| कित ||| 0.625
	   means that the probability of "kit" getting transliterated as "कित" is 0.625
	3. Configuration file for the decoder ("en-hi-translit.ini") - This file contains targets to the 
	   language model file & the phrase-translation table file. Also, different parameters like
	   translation model weight, distortion limit, language model weight are set from here.
	
	More information on these files can be found on the Moses' official website -
	http://www.statmt.org/moses/
	OR
	in the "Moses' Manual.pdf" present in this directory.
	
	To know the value of different parameters in the .ini file for the purposes of creating the trained
	models, please have a look at the final project report in "ROOT\Docs\Reports" directory.
	
b. Moses Decoder and related files
The directory "ROOT\Work" contains 3 files which are related to the Moses decoder,
	1. moses.exe - This is the Moses decoder. Type "moses.exe --help" on the command prompt
	   to know about various configurations, "n-best-list" is of great importance to us.
	2. cygwin1.dll - Moses is designed to run on Linux systems but this copy of Moses was installed on
	   a Windows machine using Cygwin. cygwin1.dll acts a Linux API emulation layer providing
	   substantial Linux API functionality.
	3. cygz.dll - Again, this file is also required to be able to run Moses.

c. Driver Program (C++ program):
There are 2 files -
	1. transliterate.cpp (in "ROOT\Work" directory)
	2. translitere.exe (in "ROOT\Work" directory) - This is compiled from tranliterate.cpp
	
Description of "transliterate.cpp"
----------------------------------
This program takes as input a file of names to be transliterated and the source language
of these names. For e.g., "transliterate.exe en-sample-input.txt en"

It uses the trained models which are stored in the "ROOT\Work\trained_models" directory to decode
the input names. A step-by-step detail is given below:
1. Clean the input file & space-delimit the characters
2. Pass it to baseline transliteration system
3. Pass it to syllabification system
4. Pass syllabification output to syllable-based transliteration system
5. From the results of baseline transliteration system and the syllable-based transliteration system,
   choose best 6 results and print them to a file.
   
This program also stores all the intermediate files generated in the "ROOT\Work\intermediate_results"
directory. Descriptions of these files will be given in SECTION (B).

This source file has the following modules (functions):-

1. void clean_input_file (); 
   - Cleans the input file, removes extra spaces, empty lines, lowercases all characters, space-delimits all characters
   
2. void perform_baseline_transliteration (); 
   - Runs baseline transliteration system decoder
   
3. void read_baseline_results (); 
   - Reads the output file of baseline transliteration system and stores in variables
   
4. void perform_syllabification ();
   - Runs syllabification system decoder
   
5. void read_syll_results ();
   - Reads the output file of syllabification system and stores in variables, also creates and input file for the syllable-based transliteration system
   
6. void perform_transliteration ();
   - Runs syllabification-based transliteraion system decoder
   
7. void read_syll_based_translit_results ();
   - Reads the output file of syllable-based transliteration system and stores in variables
   
8. void find_top6 ();
   - Reads the baseline and syllable-based transliteration system outputs, generates a list of best 6 results
  
Other Functions :-

9. string put_spaces (string)
   - Puts spaces between each character in an ANSI unicode string (English)

10. string put_spaces2 (string)
   - Puts spaces between each character in an UTF-8 unicode string (Hindi)

11. void find_and_replace (string &, string, string)
   - Finds a substring in a source string and replaced it by another substring

12. void remove_first_char (string &, string)
   - Used to remove first character if it's a maatra from a string

A flow-chart of the way program works is shown in the "Flow-chart.pdf" in the "ROOT\Work" directory.

- After making any changed to this file, compile it to "transliterate.exe" in the same directory.

---------------------------------------
SECTION (B) Output Files - Description
---------------------------------------

- The driver program generates 5 intermediate files while performing transliteration task.
- For e.g., and English to Hindi transliteration task will generate 5 intermediate files &
  1 final results file.
- The intermediate files are placed in "ROOT\Work\intermediate_file" directory
- The final results file is placed in "ROOT\Work" directory

	Intermediate files
	------------------
	a. en-clean-input.txt (in "ROOT\Work\intermediate_results" folder) - it's the cleaned-up input file, all spaces are removed, blank lines are deleted, and all characters are lowercased then all the characters are space-delimited
	b. en-hi-baseline-results.txt (in "ROOT\Work\intermediate_results" folder) - baseline results file generated by moses
	c. en-sy-results.txt (in "ROOT\Work\intermediate_results" folder) - syllabification results file generated by moses
	d. en-sy-bestresult.txt (in "ROOT\Work\intermediate_results" folder) - It's created in a format so as to serve as the input to the syllable-based transliteration system
	e. en-hi-translit-results.txt (in "ROOT\Work\intermediate_results" folder) - syllable-based transliteration results file generated by moses
		
	Final results file
	------------------
	f. en-hi-best6-results.txt (in "ROOT\Work" folder) - this is the FINAL results file, which contains the best 6 results for every input name, separated by commas
	
	An Example entry from en-hi-baseline-results.txt
	2 ||| न ल ि न  ||| d: 0 lm: -11.7434 tm: 0 -0.0563312 0 -1.87243 0.999896 w: -4 ||| -4.32169
	
	If we tokenize the above string by "|||" then
	1st token (2): It refers to input name no. The count starts from 0. So, 2 represents 3rd input name
	2nd token (न ल ि न): It is the baseline transliteration system's output transliteration.
	3rd token (d: 0 lm: -11.7434 tm: 0 -0.0563312 0 -1.87243 0.999896 w: -4)
		(a) d: 0 -> Referes to zero distortion
		(b) lm: -11.7434 -> Referes to language model score, this score is then multiplied internally
			by the weight which was specified in the .ini file (configuration file)
		(c) tm: 0 -0.0563312 0 -1.87243 0.999896 -> Refers to scores of translation model weights
		(d) w: -4 -> Refers to word penalty
		For more information on these weights, please refer to the "Moses' Manual.pdf"
	4th token (-4.32169): Overall score, a weighted sum of all the scores. This score is read in
	the program variables to choose a list of best 6 transliterations.
	

---------------------------------------
SECTION (C) Training - An Overview
---------------------------------------
As it might be required at a later stage, this section explains how new training data can be
added to this transliteration tool.

Step 1: Moses needs to be installed on the machine. For the installation of Moses, Moses offers 
a step-by-step example of downloading, compiling and running the Moses decoder and related
support tools at this link -
http://www.statmt.org/moses_steps.html

Step 2: The new training data which has to be added to the existing training data ("ROOT\Work\training-
corpora.xls") has to be manually syllabified first.

Step 3: All 6 models (as discussed above) need to be updated for the new training data.
For each model, the input format differs. These input formats are shown in the 
sub-directory "Training Formats". For each model, firstly a language model needs to be created.
And then, the translation model needs to be trained. The commands for both the operations 
is described in the step-by-step guide.

Step 4: Once all 6 models have been created, they need to be stored in the "ROOT\Work\trained_models"
directory with appropriate names. These new models should be written over the existing models. It is
advisable to take a back-up of the existing models before overwriting them. Only the language models
(.lm) and the phrase-translation tables (.gz) need to be overwritten. The .ini file (configuration file)
should not be changed.